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Executive  Summary 

The  purpose  of  this  study  was  to  conduct  a  feasibility  study  for  an  assessment  in  team 
games.  The  logic  for  both  individual  and  team  assessment  is  outlined  in  a  companion  report  to 
ONR  (O’Neil,  Baker,  Wainess,  Chen,  Mislevy,  &  Kyllonen,  2004).  The  goal  of  the  research  in 
general  was  to  produce  an  analysis  to  permit  the  development  and  application  of  metrics  to 
assess  the  impact  of  participating  in  a  team  game  environment.  The  game  environment  provides 
an  opportunity  to  assess  key  attributes  of  learning  in  teams.  These  include  both  process  and 
outcome  skills  for  teams.  Another  form  of  impact  involves  the  development  of  social  capital 
among  the  players.  A  literature  review  and  frameworks  for  the  measurement  of  team  skills  and 
social  capital  are  included.  Finally,  a  discussion  of  several  issues  related  to  the  relationship  of 
DARWARS  to  combat  readiness  completes  the  report. 
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Purpose 

The  purpose  of  this  study  was  to  conduct  a  feasibility  study  for  an  assessment  in  team 
games.  The  logic  for  both  individual  and  team  assessment  is  outlined  in  a  companion  report  to 
ONR  (O’Neil,  Baker,  Wainess,  Chen,  Mislevy,  &  Kyllonen,  2004)  and  that  text,  where 
appropriate,  is  provided  in  this  report.  The  goal  of  the  research  in  general  was  to  produce  an 
analysis  to  permit  the  development  and  application  of  metrics  to  assess  the  impact  of 
participating  in  a  team  game  environment.  The  game  environment  provides  an  opportunity  to 
assess  key  attributes  of  learning  in  teams.  These  include  both  process  and  outcome  skills  for 
teams.  Another  form  of  impact  involves  the  development  of  social  capital  among  the  players. 

While  effectiveness  of  game  environments  can  be  documented  in  terms  of  intensity  and 
longevity  of  engagement,  there  is  much  less  solid  information  about  what  outcomes  are 
systematically  achieved  by  the  use  of  multi-user  games  to  train  participants  in  acquiring 
technical  or  strategic  material.  What  is  missing  is  how  games  should  be  evaluated  at  a  number  of 
points.  First  is  the  degree  to  which  they  are  designed  to  foster  the  key  skills  and  strategies 
desired.  Secondly,  the  impact  of  game  playing  needs  to  be  studied  to  determine  what  works, 
when,  and  for  which  players.  This  report  will  support  the  design  of  an  approach  that  will  allow 
the  development  of  techniques  for  assessing  learning  of  teams  through  games. 

Literature  Review 

One  of  our  initial  tasks  was  to  conduct  a  literature  review  for  team  games  focusing  on 
empirical  studies  using  adults.  Six  search  terms  from  PsycINFO  were  used  for  the  years 
1990-2003:  multiplayer,  multi-player,  MUD,  massively  multiplayer,  massively  multi-player,  and 
MMG  OR  MMOG.  The  term  MUD  means  either  multi-user  domain  or  multi-user  dungeon. 

MMG  refers  to  massively  multiplayer  game,  and  MMOG  refers  to  massively  multiplayer  online 
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game.  As  may  be  seen  in  Table  1,  only  three  of  the  search  terms  ( multiplayer ,  multi-player,  and 
MUD)  returned  any  abstracts.  For  the  term  multiplayer  there  were  a  total  of  12  abstracts 
(PsycINFO  returned  4  unique  abstracts,  EducationAbs  returned  3  abstracts,  and  SocialSciAbs 
returned  5).  For  the  term  multi-player  there  were  a  total  of  1 1  unique  abstracts  (PsycINFO 
returned  3  unique  abstracts,  EducationAbs  returned  3,  and  SocialSciAbs  returned  5).  The  term 
MUD  returned  the  largest  number  of  abstracts  (. N  =  23;  17  unique  abstracts  from  PsycINFO,  5 
from  EducationAbs,  and  1  from  SocialSciAbs).  All  searches  combined  yielded  46  unique 
abstracts,  The  same  relevance  criteria  used  in  the  broader  video  games  search  (see  O’Neil  et  al., 
2004)  were  applied  to  this  multiplayer  video  games  search).  Only  one  article  was 
relevant— ‘Team  practice  schedules:  What  do  we  know?”  by  Lori  Rhodenizer,  Clint  A.  Bowers, 
and  Maureen  Bergondy,  published  August  1998  in  Perceptual  &  Motor  Skills  (pp.  31-34).  The 
article  reported  on  an  investigation  of  the  effect  of  practice  schedule  on  team  learning  in  a 
multiplayer  radar  simulation  decision-making  task.  In  a  design  using  32  two-person  teams, 
results  indicated  that  teams  that  practiced  under  a  distributed  schedule  performed  better  than 
teams  that  practiced  under  a  massed  schedule.  We  also  conducted  a  second  search  for  articles 
published  in  2004,  for  this  report,  and  found  none  that  met  our  criteria. 

Table  1 

Multiplayer:  Search 


Search  term 

Total  hits  per 
database 

Relevant 

hits3 

Relevant 

empirical 

abstractsb 

Relevant 
empirical 
military  abstracts0 

Multiplayer 

12 

1 

1 

0 

Multi-player 

11 

0 

0 

0 

MUD 

23 

0 

0 

0 

Massively  multiplayer 

0 

0 

0 

0 

MMOGorMMG 

0 

0 

0 

0 

“Used  in  training  and  adults.  bEither  qualitative  or  quantitative  information.  cUsed  military  personnel  as  participants. 
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In  summary,  the  literature  search  indicated  that  there  is  essentially  no  relevant  empirical 
research  conducted  on  the  topic  of  learning  or  assessment  using  team  games.  There  was  a  single 
exception  for  multiplayer  games,  and  this  study  focused  on  2-person  teams.  Thus,  there  is  very 
limited  research  evidence  on  the  training  effectiveness  of  team  games  or  the  skills  required.  We 
suggest,  based  on  our  expert  opinion,  that  the  relevant  skills  to  be  assessed  by  DARWARS 
would  be  team  skills  and  social  capital.  The  following  section  offers  a  theoretical  review  of 
measurement  of  team  skills. 

Team  Skills 

An  important  distinction  is  offered  by  Salas,  Dickinson,  Converse,  and  Tannenbaum 
(1992)  for  types  of  team  skills,  that  is,  task  work  versus  teamwork.  Task  work  skills  influence 
how  well  an  individual  performs  on  a  particular  team  task,  whereas  teamwork  skills,  or  team 
process  skills,  influence  how  effective  team  members  will  be  as  part  of  a  team.  A  theoretical 
framework  has  been  provided  by  Dr.  Jan  Cannon-Bowers  (Cannon-Bowers  &  Bowers,  2004).  As 
part  of  another  ONR  contract,  for  which  Dr.  O’Neil  is  the  PI,  Cannon-Bowers  has  produced  a  set 
of  guidelines  to  teach  and  assess  team  skills.  Table  2  provides  a  list  of  her  team  performance 
measurement  guidelines.  Tables  3  and  4  provide  two  examples  of  the  guidelines.  It  is 
recommended  for  DARWARS  team  performance  assessment  that  metrics  be  developed  and  an 
evaluation  study  using  the  four  levels  of  Kirkpatrick’s  (1994)  evaluation  framework  be 
conducted  using  her  team  performance  guidelines  as  design  standards. 
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Table  2 

Team  Performance  Measurement  (Cannon-Bowers  &  Bowers,  2004) 

1.  Assess  pre-training  teamwork  competencies  (i.e.,  knowledge,  skills  and  attitudes) 

2.  Construct  team-level  outcome  measures  for  use  at  the  conclusion  of  training  (e.g., 
mission  accomplishment,  sales,  safety  records,  etc.) 

3.  Construct  measures  of  transfer  performance  (on  the  job) 

4.  Construct  teamwork  process  measures  (diagnostic;  during  training) 

5.  Link  measurement  points  to  training  events  in  scenarios/vignettes/role  plays 

6.  Record  as  much  as  possible  (voice,  keystrokes,  etc.)  during  training  sessions 

7.  Use  checklists  and  other  observational  devices  to  aid  collection  of  performance  data 

8.  Foster  systematic  intra-team  monitoring  and  assessment  during  training  and  debrief 
sessions 

9.  Assess  post-training  knowledge  (e.g.,  interpositional  knowledge;  shared  knowledge, 
shared  mental  models,  shared  expectations) 

10.  Assess  post-training  skills  (e.g.,  communication  skills;  intra  team  monitoring,  error 
detection,  correction;  giving/receiving  feedback,  self-correction  strategies;  backup, 
compensatory  behaviors;  team  leader  behavior) 

11.  Assess  post-training  attitudes  (e.g.,  collective  efficacy) 
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Table  3 

Assess  Post-Training  Knowledge  (e.g.,  Interpositional  Knowledge;  Shared  Knowledge,  Shared  Mental  Models, 
Shared  Expectations)  (Cannon-Bowers  &  Bowers,  v.l,  9/17/04)  (Cannon-Bowers  &  Bowers,  2004) 


1.  Guideline  (nontechnical):  Develop  measures  of  interpositional  knowledge  and  shared  knowledge  to  use  at 

the  conclusion  of  training;  these  can  be  compared  to  pre-training  measures  if 
available. 

2.  Guideline  (technical):  If  team  training  is  successful,  team  members  should  have  a  good  sense  of  their 

own  role  in  the  team  as  well  as  the  role  of  teammates  (i.e.,  interpositional 
knowledge),  and  shared  crucial  knowledge  about  how  to  perform  (i.e.,  shared 
mental  models).  These  should  be  measured  at  the  conclusion  of  training. 

3.  Guideline  based  on:  Research 


4.  Degree  of  confidence:  High 

5.  Comments:  Research  suggests  that  team  members  must  understand  their  role  in  the  team  as 

well  as  the  role  of  teammates.  This  knowledge  enables  them  to  better 
coordinate  during  episodes  of  performance.  Relatively  simple  measures  of 
interpositional  knowledge  can  be  useful  in  assessing  this  competency.  For 
example,  paper  and  pencil  measures  that  ask  team  members  to  rate  the  tasks  that 
they  and  their  teammates  are  responsible  for  have  been  used  for  this  purpose. 

Shared  mental  models— that  is  knowledge  that  the  team  must  share  so  that  they 
can  adapt  their  performance  quickly  and  without  much  discussion— have  been 
found  to  be  important  to  effective  team  functioning.  Several  approaches  to 
measuring  shared  mental  models  in  teams  have  been  developed. 

6.  References:  Cooke,  N.  J.,  Kiekel,  Preston  A.,  Salas,  E.,  &  Cannon-Bowers,  J.  A.  (2003). 

Measuring  team  knowledge:  A  window  to  the  cognitive  underpinnings  of  team 
performance.  Group  Dynamics,  7(3),  179-199. 


Eby,  L.  T.,  Meade,  A.  W.,  &  Parisi,  A.  G.  (1999).  The  development  of  an 
individual-level  teamwork  expectations  measure  and  the  application  of  a  within- 
group  agreement  statistic  to  assess  shared  expectations  for  teamwork. 
Organizational  Research  Methods,  2,  366-394. 

Hoeft,  R.  M.,  Jentsch,  F.  G.,  &  Harper,  M.  E.  (2003).  TPL-KATS— Concept 
Map:  A  computerized  knowledge  assessment  tool.  Computers  in  Human 
Behavior,  19,  653-657. 


Kraiger,  K.,  &  Wenzel,  L.  H.  (1997).  Conceptual  development  and  empirical 
evaluation  of  measures  of  shared  mental  models  as  indicators  of  team 
effectiveness.  In  M.  T.  Brannick,  E.  Salas,  &  C.  Prince  (Eds.),  Team 
performance  assessment  and  measurement:  Theory,  methods,  and  applications 
(pp.  63-84).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 

Neuman,  G.  A.,  &  Wright,  J.  (1999).  team  effectiveness:  beyond  skills  and 
cognitive  ability.  Journal  of  Applied  Psychology,  84,  376-389. 
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Table  4 

Assess  Post-Training  Skills  (e.g.,  Communication  Skills;  Intra-Team  Monitoring,  Error  Detection,  Correction; 
Giving/Receiving  Feedback,  Self-Correction  Strategies;  Backup,  Compensatory  Behaviors;  Team  Leader 
Behavior)  (Cannon-Bowers  &  Bowers,  v.l,  9/17/04)  (Cannon-Bowers  &  Bowers,  2004) 


1.  Guideline  (nontechnical): 

Develop  measures  of  important  teamwork  skills  (identified  by  the  team  task 
analysis  conducted  prior  to  training)  as  a  means  to  assess  the  effectiveness  of 
training  interventions. 

2.  Guideline  (technical): 

Effective  teamwork  is  multidimensional,  requiring  several  team-level  skills. 
Measures  of  skills  identified  as  crucial  in  the  team  task  analysis  should  be 
developed  to  assess  training  effectiveness. 

3.  Guideline  based  on: 

Research 

4.  Degree  of  confidence: 

High 

5.  Comments: 

Research  indicates  that  teams  require  several  team-level  skills,  depending  on  the 
nature  of  the  team  task  they  are  performing  (Cannon-Bowers,  et  al.,  1995).  The 
purpose  of  a  team  task  analysis  (conducted  prior  to  training  design)  is  to 
determine  which  skills  are  most  important  to  the  team  at  hand.  Once  the  crucial 
team  skills  are  identified,  measures  of  these  skills  must  be  developed  for  use  at 
the  conclusion  of  training.  The  most  typical  team  skills  include:  communication 
skills;  intra-team  monitoring,  error  detection,  correction;  giving/receiving 
feedback,  self-correction  strategies;  backup,  compensatory  behaviors;  team 
leader  behavior. 

While  traditional  (paper  and  pencil)  measures  of  team  skills  can  be  used,  a  better 
strategy  is  to  develop  work  samples,  exercises  or  scenarios  for  team  members  to 
perform.  Such  measures  are  typically  better  because  they  sample  actual 
behavior,  which  is  affected  by  many  factors  including  the  team  members’  ability 
to  identify  when  and  how  to  respond  given  situational  demands.  If  behavior- 
based  measures  are  used,  observational  checklists  and  other  devices  can  be  used 
to  assess  whether  the  team  has  acquired  the  skill. 

6.  References: 

Cannon-Bowers,  J.  A.,  Tannenbaum,  S.  I.,  Salas,  E.,  &  Volpe,  C.  (1995). 

Defining  competencies  and  establishing  team  training  requirements.  In  R.  A. 
Guzzo  &  E.  Salas  (Eds.),  Team  effectiveness  and  decision-making  in 
organizations  (pp.  333-380).  San  Francisco,  CA:  Jossey-Bass. 

Dickenson,  T.  L.,  &  McIntyre,  R.  M.  (1997).  A  conceptual  framework  for  team 
performance  measurement.  In  M.  T.  Brannick,  E.  Salas,  &  C.  Prince  (Eds.), 

Team  performance  assessment  and  measurement:  Theory,  methods,  and 
applications  (pp.  19-44).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 

Johnston,  J.  H.,  Smith-Jentsch,  K.  A.,  &  Cannon-Bowers,  J.  A.  (1997). 
Performance  measurement  tools  for  enhancing  team  decision-making 
training.  In  M.  T.  Brannick,  E.  Salas,  &  C.  Prince  (Eds.),  Team  performance 
assessment  and  measurement:  Theory,  methods,  and  applications  (pp.  311- 
327).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 

Social  Capital 

It  could  be  argued  that  DARWARS  should  increase  the  social  capital  of  the  DoD.  Social 
capital  is  one  of  the  invisible  assets  of  an  organization  and  complements  its  intellectual. 
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financial,  and  physical  assets  (Hargreaves,  2003).  As  may  be  seen  in  Figure  1,  social  capital 
consists  of  trust,  collective  efficacy,  collective  effort,  teamwork,  transparency,  and  networks 
between  its  members  and  its  individual  partners.  It  is  recommended  that  DARWARS  also  be 
evaluated  using  this  framework. 


Figure  1.  Elements  of  social  capital. 


Measurement  of  social  capital  is  in  its  infancy.  In  prior  research,  we  have  developed 
survey  measures  of  some  of  the  constructs  in  Figure  1.  The  Teamwork  Questionnaire  (O’Neil, 
Wang,  Lee,  Mulkey,  &  Baker,  2003)  has  been  used  in  several  studies,  has  acceptable  reliability 
and  validity,  and  measures  teamwork  skills  (i.e.,  adaptability,  coordination,  decision  making, 
leadership,  and  interpersonal  teamwork  skills).  We  have  also  developed  survey  measures  of 
collective  efficacy  and  effort  (Marshall  et  al.,  2003).  The  Marshall  et  al.  set  of  measures  (which 
includes  the  teamwork  measure)  was  recently  used  in  a  healthcare  setting  as  the  Healthcare 
Teams  Questionnaire.  We  have  attached  a  first-cut  adaptation  of  that  questionnaire  in  the 
Appendix.  The  item  numbers  track  the  scoring  key.  For  a  DARWARS  application,  one  would 
need  to  select  or  develop  items  to  measure  trust,  networking  and  transparency,  as  well  as  a  set  of 
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measures  to  target  specific  DARWARS  functionality  (e.g.,  diagnosis,  individualized  training 
plans,  scheduling  of  assets,  multiplayer  games).  We  have  also  used  a  survey  approach  in  the  past 
for  measurement  of  Information  Technology  (IT)  systems  (Muraida,  Spector,  O’Neil,  &  Marlino, 
1993).  A  recommended  set  of  DARWARS  functionality  survey  questions  to  be  developed  would 
focus  on  measurement  of  buy-in  and  the  perceptions  of  success  and  failure  of  DARWARS. 
Elements  would  include  how  one  thinks  and  feels  about  DARWARS. 

This  effort  should  also  assess  over  time  the  development  of  social  capital  by  DARWARS 
participants.  Collaborative  decision  making  with  DARPA  should  occur  regarding  the  design, 
comparative  analysis,  frequency  of  measurement,  and  reporting  framework.  The  effort  will  be 
divided  into  two  major  activities:  (A)  design  and  initial  administration,  and  (B)  implementation 
of  successive  survey  waves,  analysis,  and  reporting. 

A.  Design  and  Initial  Administration 

Objective:  To  design  and  field  the  first  administration  of  an  on-line  survey  to  evaluate 
the  DARWARS  impact. 

Task  1.  Create  measures  of  the  components  of  social  capital  (i.e.,  collective  efficacy, 
teamwork,  collective  effort,  transparency,  networks,  and  trust)  and  augment  the  item 
pool  so  that  at  least  15  items  for  each  component  can  be  administered.  In  addition, 
survey  items  will  be  created  regarding  the  functional  components  of  DARWARS. 

B.  Implementation  of  Successive  Survey  Waves,  Analysis,  and  Reporting 
Objective:  Implement  the  assessment  on  a  regular  basis 

Task  2.  Revise  scales  as  necessary,  administer  the  survey,  analyze  data,  and  report. 

Relationship  of  DARWARS  and  Combat  Readiness 

In  order  to  estimate  the  potential  success  or  failure  of  DARWARS  on  readiness  several 
issues  would  need  to  be  resolved,  (a)  Alignment  of  DARWARS  training  philosophy  of 
experiential  training  with  a  Service  mission  philosophy  that  embeds  individual  training  standards 
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within  collective  training  standards.  These  standards  are  embedded  in  mission-essential  task  lists, 
and  finally  all  standards  contribute  to  the  mission  (PM  Training  Systems,  n.d.,  p.  1).  (b)  A 
concept  of  operation  of  how  DARWARS  would  be  used,  either  on-duty  or  off-duty.  For 
example,  on-duty  implementation  issues  are  shown  in  Table  5,  and  DARWARS  off-duty  training 
issues  are  shown  in  Table  6.  The  nature  of  the  DARWARS  impact  on  readiness  would  differ 
depending  on  whether  DARWARS  is  implemented  on-duty  or  off-duty.  The  same  “game”  would 
not  be  implemented  in  both  environments. 

Table  5 

On-Duty  Implementation  Issues:  Impact  on  Game 

If  on-duty  in  military  school: 

-  Integrate  into  Plan  of  Instruction 

•  Specific  objectives,  assessments,  minimal  role  playing 

-  Instructor  training 

•  Need  nature/type  of  feedback 

-  Doctrinal  accuracy 

•  Limited  role  for  fantasy 

-  Motivation  provided  by  instructor 

•  Game  can  be  less  fun 

-  Good  applications 

•  Initial  acquisition  training 

-  Time  available  is  expressed  in  hours/days/weeks 


Table  6 

Off-Duty  Implementation  Issues:  Impact  on  Game 
If  off-duty: 

-  Integrate  into  recreation/family  time 

•  Objectives/assessments  less  important 

•  Flexible  role  playing 

-  Student  “training”  need 

•  Improve  self-regulation 

•  Increase  requirement  for  feedback,  automated  After-Action  Reviews 

-  Flexible  Doctrine/Strategy/Tactics 

-  Motivation  provided  by  student/game 

•  Game  needs  to  be  fun 

-  Good  application 

•  Refresher  training  to  prevent  skill  decay 

-  Time  available  is  expressed  in  minutes/hours 
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(c)  An  adoption  of  a  framework  of  training  readiness  and  assessable  indicators.  Orlansky, 
Hammon,  and  Horowitz  (1997)  have  outlined  a  framework  and  indicators.  First,  citing  Finch 
(1996),  they  define  readiness: 

Readiness  is,  as  the  Deputy  Undersecretary  of  Defense  for  Readiness  describes  it: 

the  general  ability  of  forces  to  arrive  where  they  are  needed,  on  time  and  prepared  to 
effectively  carry  out  assigned  mission  objectives  for  which  they  were  designed.  The 
ability  of  units  to  be  ready  on  time  to  carry  out  their  missions,  in  turn,  is  a  function  of 
having  the  equipment,  supplies,  logistics  and  experienced  people  with  the  skills  to 
accomplish  assigned  tasks.  (Finch,  1996).  (as  quoted  by  Orlansky,  Hammon,  & 
Horowitz,  1997,  p.  S-l) 

As  may  be  seen  in  Table  7,  Orlansky  et  al.  provide  an  input,  process,  and  output  to  further 
analyze  readiness.  We  have  adapted  one  of  their  tables. 


Table  7 

Training  Process  Indicators 


Input 

Process 

Output 

Resources 

Training 

Performance 

Costs 

Courses 

Measures  of  effectiveness 

Events 

Exercise  outcomes 

Exercised 

Performance  to  standards 

OFTEMPO 

Accomplishments 

Training  readiness 

Source:  Adapted  from  Orlansky,  Hammon,  &  Horowitz,  1997,  p.  S-2. 


Orlansky  et  al.  (1997)  also  make  a  critical  distinction  between  type  of  indicator— i.e., 
demonstrated  training  performance  (outcome  measures)— and  training  accomplishment  (process 
measures).  As  may  be  seen  in  Table  8,  each  major  type  of  indicator  has  multiple  indicators.  For 
DARWARS,  choices  would  have  to  be  made  among  indicators  depending  on  the  specific 
DARWARS  features;  for  those  that  are  Service-specific,  one  would  accept  that  Service’s 
indicator  of  either  training  performance  or  training  accomplishment. 
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Table  8 

Training  Indicators 


Type  of  indicator 

Indicator 

Service 

Demonstrated  training  performance 

percent  of  crews  or  platoons  qualified 

Army 

Marine  Corps 

percent  of  submarine  Training  Readiness 
Examination  above/below  average 

Navy  subsurface 

percent  of  Operational  Readiness 

Inspection  excellent  or  outstanding 

Air  Force 

percent  of  tasks  trained  to  standard 

Army 

Marine  Corps 

Training  accomplishment 

percent  of  mission  essential  tasks  trained 

Army 

Marine  Corps 

percent  training  accomplished  by  primary 
mission  areas 

Navy 

percent  of  training  accomplished  (percent 
crews  combat  ready) 

Navy  aviation 
USMC  aviation 

percent  Graduated  Combat  Capability 
level  B  or  A 

Air  Force 

percent  participation  in  Combat  Training 
Centers/Combat  Arms  Exercises 

Army 

Marine  Corps 

Source:  Orlansky,  Hammon,  &  Horowitz,  1997,  p.  S-5. 


Summary 

The  purpose  of  this  study  was  to  conduct  a  feasibility  study  for  an  assessment  in  team 
games.  The  logic  for  both  individual  and  team  assessment  is  outlined  in  a  companion  report  to 
ONR  (O’Neil,  Baker,  Wainess,  Chen,  Mislevy,  &  Kyllonen,  2004).  The  goal  of  the  research  in 
general  was  to  produce  an  analysis  to  permit  the  development  and  application  of  metrics  to 
assess  the  impact  of  participating  in  a  team  game  environment.  The  game  environment  provides 
an  opportunity  to  assess  key  attributes  of  learning  in  teams.  These  include  both  process  and 
outcome  skills  for  teams.  Another  form  of  impact  involves  the  development  of  social  capital 
among  the  players.  A  literature  review  and  frameworks  for  the  measurement  of  team  skills  and 
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social  capital  are  included.  Finally,  a  discussion  of  several  issues  related  to  the  relationship  of 
DARWARS  to  combat  readiness  completes  the  report. 
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Appendix 

DARWARS  Social  Capital  Questionnaire 


Revision  6/23/04  OF  HEALTHCARE  TEAMS  QUESTIONNAIRE 

NOTE:  Three  scales  in  the  previous  version  have  been  deleted  from  the  current  revision 
(6/23/04).  Items  will  be  renumbered  in  the  next  revision.  Item  stems  need  to  be  tailored  to 
DARWARS. 

SCORING  KEY 

DARWARS  SOCIAL  CAPITAL  QUESTIONNAIRE 


Scales 

Collective  Efficacy  ( N  =  7) 
Collective  Effort  (/V  =  8) 
Coordination  ( N  =  5) 
Decision  Making  (N  =  6) 
Leadership  (N  =  7) 
Interpersonal  ( N  =  6) 
Adaptability  (N  =  5) 
Communication  (N=7) 


Items 

3, 9, 17, 27, 38, 48, 55 

I,  4,  10, 20,  30, 41, 49, 57 
16,  24,  32, 42,  56 

II,  18,25,  34,43,50 
5, 12, 19, 26,  36, 45, 52 
14, 21, 28,  37, 58, 62 
29, 39, 46, 53, 61 
8,23,31,40,47, 54,60 
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Collective  Efficacy  Trait  Subscale  (L.  Marshall-Stern  &  H.  F.  O’Neil,  2001) 

3.  I  believe  that  my  team  will  contribute  to  our  team's  patient  satisfaction  ratings. 

9.  I’m  certain  that  my  team  has  knowledge  of  how  to  work  in  a  team . 

17.  I’m  confident  my  team  has  the  basic  teamwork  skills. 

27.  I’m  confident  my  team  can  coordinate  teamwork  activities. 

38.  I’m  confident  my  team  can  do  an  excellent  job  on  assignments  and  tasks. 

48.  I’m  certain  my  team  has  excellent  patient  care  task-related  skills. 

55.  I’m  confident  my  team  will  do  well  on  our  work. 

Collective  Effort  (L.  Marshall-Stern  &  H.  F.  O’Neil,  2001) 

1.  My  team  uses  methods  and  procedures  for  working  together  that  are  just  right  for  the  tasks 
we  have  to  perform. 

4.  My  team  works  hard  to  do  well  even  if  they  don’t  like  a  task 

10.  My  team  puts  forth  it’s  best  effort  on  tasks. 

20.  My  team  works  as  hard  as  possible  on  tasks. 

30.  My  team  concentrates  as  hard  as  they  can  when  doing  a  task. 

41.  My  team  works  hard  on  a  task  even  if  it  does  not  count. 

49.  My  team  is  willing  to  do  extra  work  on  tasks  to  improve  our  knowledge. 

57.  My  team  believes  practice  makes  perfect 

COORDINATION  -  Organizing  team  activities  to  complete  a  task  on  time 
16.  When  I  work  as  part  of  a  team,  I  allocate  the  tasks  according  to  each  team  member’s 
abilities. 

24.  When  I  work  as  part  of  a  team,  I  help  ensure  the  proper  balancing  of  the  workload. 

32.  When  I  work  as  part  of  a  team,  I  do  my  part  of  the  organization  in  a  timely  manner. 

42.  When  I  work  as  part  of  a  team,  I  track  other  team  members’  progress. 

56.  When  I  work  as  part  of  a  team,  I  emphasize  the  meeting  of  deadlines. 


DECISION  MAKING  -  Using  available  information  to  make  decisions 

11.  When  I  work  as  part  of  a  team,  I  understand  and  contribute  to  the  organizational  goals. 

18.  When  I  work  as  part  of  a  team,  I  know  the  process  of  making  a  decision. 

25.  When  I  work  as  part  of  a  team,  I  know  how  to  weigh  the  relative  importance  among 
different  issues. 

34.  When  I  work  as  part  of  a  team,  I  prepare  sufficiently  to  make  a  decision. 

43.  When  I  work  as  part  of  a  team,  I  solicit  input  for  decision  making  from  my  team  members. 

50.  When  I  work  as  part  of  a  team,  I  am  able  to  change  decisions  based  upon  new  information. 
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LEADERSHIP  -  Providing  direction  for  the  team 
5.  When  I  work  as  part  of  a  team,  I  exercise  leadership. 

12.  When  I  work  as  part  of  a  team,  I  teach  other  team  members. 

19.  When  I  work  as  part  of  a  team,  I  serve  as  a  role  model  in  formal  and  informal  interactions. 
26.  When  I  work  as  part  of  a  team,  I  lead  when  appropriate,  mobilizing  the  group  for  high 
performance. 

36.  When  I  work  as  part  of  a  team,  I  lead  the  team  effectively. 

45.  When  I  work  as  part  of  a  team,  I  demonstrate  leadership  and  ensure  team  results. 

52.  When  I  work  as  part  of  a  team,  I  try  to  bring  out  the  best  in  others. 

INTERPERSONAL  -  Interacting  cooperatively  with  other  team  members 

14.  When  I  work  as  part  of  a  team,  I  interact  cooperatively  with  other  team  members. 

21.  When  I  work  as  part  of  a  team,  I  conduct  myself  with  courtesy. 

28.  When  I  work  as  part  of  a  team,  I  respect  the  thoughts  and  opinions  of  others  in  the  team. 

37.  When  I  work  as  part  of  a  team,  I  treat  others  with  courtesy. 

58.  When  I  work  as  part  of  a  team,  I  accept  individual  differences  among  members. 

62.  When  I  work  as  part  of  a  team,  I  treat  all  my  team  members  as  equals. 

ADAPTABILITY  -  Recognizing  problems  and  responding  appropriately 

29.  When  I  work  as  part  of  a  team,  I  can  identify  potential  problems  readily. 

39.  When  I  work  as  part  of  a  team,  I  willingly  contribute  solutions  to  resolve  problems. 

46.  When  I  work  as  part  of  a  team,  I  adapt  readily  to  varying  conditions  and  demands. 

53.  When  I  work  as  part  of  a  team,  I  recognize  conflict. 

61.  When  I  work  as  part  of  a  team,  I  identify  needs  or  requirements  and  develop  quality/timely 
solutions. 

COMMUNICATION  -  Clear  and  accurate  exchange  of  information 
8.  When  I  work  as  part  of  a  team,  I  ensure  the  instructions  are  understood  by  all  team 
members  prior  to  starting  the  task. 

23.  When  I  work  as  part  of  a  team,  I  ask  for  the  instructions  to  be  clarified  when  it  appears  not 
all  the  team  members  understand  the  task. 

31.  When  I  work  as  part  of  a  team,  I  communicate  in  a  manner  to  ensure  mutual  understanding. 

40.  When  I  work  as  part  of  a  team,  I  seek  and  respond  to  feedback. 

47.  When  I  work  as  part  of  a  team,  I  listen  attentively. 

54.  When  I  work  as  part  of  a  team,  I  clearly  and  accurately  exchange  information. 

60.  When  I  work  as  part  of  a  team,  I  pay  attention  to  what  others  are  saying. 
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PART  II  QU ESTIQNNAIRE 


Directions:  This  set  of  questions  is  to  help  us  understand  the  way  you  think  and  feel  about  working  with  others.  We 
know  that  different  parts  of  your  life,  such  as  your  job,  recreational  activities,  or  service  to  your  community,  may 
involve  working  with  others  and  have  different  requirements,  and  that  you  may  react  differently  in  each  kind  of 
activity.  Nonetheless,  read  each  statement  below  and  indicate  how  you  generally  think  or  feel.  There  are  no  right  or 
wrong  answers.  Do  not  spend  too  much  time  on  any  one  statement.  Remember,  give  the  answer  that  seems  to 
describe  how  vou  generally  think  or  feel. 


Almost 

never 

Sometimes 

Often 

Almost 

always 

My  team  uses  methods  and  procedures  for  working 
together  that  are  just  right  for  the  tasks  we  have  to 
perform. 

1 

2 

3 

4 

I  believe  that  my  team  will  contribute  to  our  team's 
patient  satisfaction  ratings. 

1 

2 

3 

4 

My  team  works  hard  to  do  well  even  if  they  don’t  like  a 
task 

1 

2 

3 

4 

When  I  work  as  part  of  a  team,  I  exercise  leadership. 

1 

2 

3 

4 

When  I  work  as  part  of  a  team,  I  ensure  the  instructions 
are  understood  by  all  team  members  prior  to  starting  the 
task. 

1 

2 

3 

4 

I’m  certain  that  my  team  has  knowledge  of  how  to  work 
in  a  team. 

1 

2 

3 

4 

My  team  puts  forth  it’s  best  effort  on  tasks. 

1 

2 

3 

4 

When  I  work  as  part  of  a  team,  I  understand  and 
contribute  to  the  organizational  goals. 

1 

2 

3 

4 

When  I  work  as  part  of  a  team,  I  teach  other  team 
members. 

1 

2 

3 

4 

When  I  work  as  part  of  a  team,  I  interact  cooperatively 
with  other  team  members. 

1 

2 

3 

4 

16.  When  I  work  as  part  of  a  team,  I  allocate  the  tasks  12  3  4 

according  to  each  team  member’s  abilities. 

17.  I’m  confident  my  team  has  the  basic  teamwork  skills.  12  3  4 

18.  When  I  work  as  part  of  a  team,  I  know  the  process  of  12  3  4 

making  a  decision. 
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19.  When  I  work  as  part  of  a  team,  I  serve  as  a  role  model  in 
formal  and  informal  interactions. 

20.  My  team  works  as  hard  as  possible  on  tasks. 

21.  When  I  work  as  part  of  a  team,  I  conduct  myself  with 
courtesy. 

23.  When  I  work  as  part  of  a  team,  I  ask  for  the  instructions 
to  be  clarified  when  it  appears  not  all  the  team  members 
understand  the  task. 

24.  When  I  work  as  part  of  a  team,  I  help  ensure  the  proper 
balancing  of  the  workload. 

25.  When  I  work  as  part  of  a  team,  I  know  how  to  weigh  the 
relative  importance  among  different  issues. 

26.  When  I  work  as  part  of  a  team,  I  lead  when  appropriate, 
mobilizing  the  group  for  high  performance. 

27.  I’m  confident  my  team  can  coordinate  teamwork  activities. 

28.  When  I  work  as  part  of  a  team,  I  respect  the  thoughts  and 
opinions  of  others  in  the  team. 

29.  When  I  work  as  part  of  a  team,  I  can  identify  potential  problems 
readily. 

30.  My  team  concentrates  as  hard  as  they  can  when  doing  a  task. 

31.  When  I  work  as  part  of  a  team,  I  communicate  in  a 
manner  to  ensure  mutual  understanding. 

32.  When  I  work  as  part  of  a  team,  I  do  my  part  of  the 
organization  in  a  timely  manner. 

34.  When  I  work  as  part  of  a  team,  I  prepare  sufficiently  to 
make  a  decision. 

36.  When  I  work  as  part  of  a  team,  I  lead  the  team  effectively. 

37.  When  I  work  as  part  of  a  team,  I  treat  others  with 
courtesy. 

38.  I’m  confident  my  team  can  do  an  excellent  job  on  the 
assignments  and  tasks. 


Almost  Almost 

never  Sometimes  Often  always 


1 

1 

1 


2 

2 

2 


3  4 

3  4 

3  4 


1  2  3 


1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 


4 


4 

4 

4 

4 

4 

4 

4 


1 


2 


3 


4 


12  3  4 

12  3  4 

1  2  3  4 

1  2  3  4 


1 


2 
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Almost 

Almost 

never 

Sometimes 

Often 

always 

39.  When  I  work  as  part  of  a  team,  I  willingly  contribute 
solutions  to  resolve  problems. 

1 

2 

3 

4 

40.  When  I  work  as  part  of  a  team,  I  seek  and  respond  to 
feedback. 

1 

2 

3 

4 

41.  My  team  works  hard  on  a  task  even  if  it  does  not  count. 

1 

2 

3 

4 

42.  When  I  work  as  part  of  a  team,  I  track  other  team 
members’  progress. 

1 

2 

3 

4 

43.  When  I  work  as  part  of  a  team,  I  solicit  input  for  decision 
making  from  my  team  members. 

1 

2 

3 

4 

45.  When  I  work  as  part  of  a  team,  I  demonstrate  leadership 
and  ensure  team  results. 

1 

2 

3 

4 

46.  When  I  work  as  part  of  a  team,  I  adapt  readily  to  varying 
conditions  and  demands. 

1 

2 

3 

4 

47.  When  I  work  as  part  of  a  team,  I  listen  attentively. 

1 

2 

3 

4 

48.  I’m  certain  my  team  has  excellent  patient  care  task- 
related  skills. 

1 

2 

3 

4 

49.  My  team  is  willing  to  do  extra  work  on  tasks  to  improve 
our  knowledge. 

1 

2 

3 

4 

50.  When  I  work  as  part  of  a  team,  I  am  able  to  change 
decisions  based  upon  new  information. 

1 

2 

3 

4 

52.  When  I  work  as  part  of  a  team,  I  try  to  bring  out  the  best 
in  others. 

1 

2 

3 

4 

53.  When  I  work  as  part  of  a  team,  I  recognize  conflict. 

1 

2 

3 

4 

54.  When  I  work  as  part  of  a  team,  I  clearly  and  accurately 
exchange  information. 

1 

2 

3 

4 

55.  I  am  confident  my  team  will  do  well  on  our  work. 

1 

2 

3 

4 

56.  When  I  work  as  part  of  a  team,  I  emphasize  the  meeting 
of  deadlines 

1 

2 

3 

4 

57.  My  team  believes  practice  makes  perfect 

1 

2 

3 

4 

58.  When  I  work  as  part  of  a  team,  I  accept  individual 
differences  among  members. 

1 

2 

3 

4 
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Almost  Almost 

never  Sometimes  Often  always 


60.  When  I  work  as  part  of  a  team,  I  pay  attention  to  what  12  3  4 

others  are  saying. 

61.  When  I  work  as  part  of  a  team,  I  identify  needs  or  12  3  4 

requirements  and  develop  quality/  timely  solutions. 

62.  When  I  work  as  part  of  a  team,  I  treat  all  my  team  12  3  4 

members  as  equals. 


Thank  you... 
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